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The Impact of Faculty Incentive Grants 
on Teaching Effectiveness 



I) Introduction. 

In recent years many American colleges and 
universities have instituted faculty incentive grants to 
reward excellent teachers. Such grants are often linked 
directly to classroom performance rather than to research 
productivity . The Handbook of Research on Teaching 
(Wittrock, 1986) reports that "^these grants are usually 
aw.^rded competitively paralleling grants awarded in 
support of research, and are seen as incentives promoting 
interest in teaching" (p. 772). J. A. Centra (1978) 
found that 58% of 756 institutions surveyed had some type 
of grant program to support development projects . 
Despite their widespread use, little attention has been 
given to evaluation of these programs. The Handbook of 
Research on Teaching notes that these types of grants 
apparently. . . are different from research grants 
at the evaluation and reporting stages because 
Levinson- Rose and Menges (1981) could find only one 
report of a study of the effectiveness ot such grant 
schemes in improving teaching. Presumably, many 
reports on such projects do not include data on 
improving teaching (p. 772). 
The one study cited (Kozma, 1978) focused on the impact 
on a single, very specific classroom function, concluding 
that even small grants could lead to an increased use of 
instructional innovations. 'The present study is an 
attempt to determine if there is any indication that the 
implementation of a faculty incentive grant progreun has 
an impact on more general measures of teaching 
effectiveness. This project is broadly structured along 
the lines of the case study method. According to A. 
Patrick Allen (1988) there is -extensive agreement- that 
this is the best framework for the evaluation of faculty 
development types of programs. (While it could be argued 
that there is a difference between faculty development 
programs proper and faculty incentive and reward systems, 
the parallels seem strong enough to suggest that this is 
an appropriate method of evaluation.) 

The focus of this study is a faculty incentive 
program at Messiah College, a private liberal arts 
institution of about 2000 students in Pennsylvania. 
Several years ago the Board of Trustees at the college 
attempted to introduce a merit pay program based on 
excellence in classroom teaching. A variety of 
administrative problems short-circuited that program 



before it could be fully implemented. As an alternative, 
a program was developed that is known as the •'Excellence 
in Teaching" Program. Funded by a private donor, four 
awards of up to $5000 each are annually presented to 
faculty. Faculty members may apply for the award based 
on student evaluations of teaching performance during the 
preceding calendar year. Additionally, applicants must 
provide evidence of effective advising, submit a course 
evaluation that has been completed by a fellow faculty 
member, and have a letter from the department chair 
indicating quality performance in both teaching and 
advising. Faculty who apply for the award must also 
submit a proposal for how the award money will be used, 
which may include travel, summer salary for research, or 
any ••expanded professional development project . •• 
Proposals need not be limited to the faculty member's 
teaching area. The listed criteria for determining which 
applicants receive the award are (1) quality of the 
proposal, (2) likelihood of successful completion of the 
project, (3) degree of enhancement for the faculty member 
involved, and (4) teaching evaluations. Award decisions 
are made by a committee consisting of the academic dean, 
assistant academic dean, and a faculty representative 
selected by the dean. Tne present study uses both 
quantitative and qualitative means to evaluate that award 
system. 

II) Procedures and Sample. 

The data for this study were obtained exclusively 
from the faculty and the academic office of Messiah 
College. Data for statistical analysis were drawn from 
the faculty evaluation instrument developed at Kansas 
State University, the Instructional Development and 
Effectiveness Assessment (IDEA), -a highly respected and 
popular student rating system in use at over 100 colleges 
and universities- (Ory, 1985). Each faculty member at 
Messiah College is required to have two courses each 
seitester evaluated via the IDEA system. 

A) Institutional Ratings . Summary institutional 
datci for twenty instructional categories provided to 
Messiah College by the IDEA system were interpreted to 
see whether there had been a change in teacher 
effectiveness ratings during the years the grant program 
has been in effect. The institutional data were obtained 
by aggregating data from evexry Messiah College class 
which had been evaluated in a given term. Cashin and 
Noma suggest that -for institutions which use the IDEA 
system for two or more years on representative samples 
of the same academic unit, the means in the Institutional 
Summaries can be compared as one measure of whether 
teaching effectiveness has changed- (1983, p. 22). 
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B) Individual Ratings. The second portion of this 
study recjuired comparison of individual teaching 
effectiveness measures during the time period the grant 
program has been in effect. Effectiveness ratings (seven 
summary outcome measures for classes evaluated by the 
IDEA system) for the semester before each award 
announcement were compared to ratings the semester after 
each award announcement. Ratings for three groups were 
analyzed: faculty who had applied for and received a 
grant (award recipients, n ^ 12), faculty who had applied 
for but not received a grant (non-recipients, n = 10), 
and faculty who had not applied in any year (control 
group, 11= 30) . 

C) Qualitative Data. Qualitative data were drawn 
from ten faculty interviews which solicited impressions 
of the grant program. This group equalled approximately 
ten percent of the full-time faculty at the college, but 
they are not necessarily representative of that group. 
The researcher made an attempt to include a spectnun of 
faculty interef ts , disciplines , lengths of employment, 
and rank^ A number of award winners purposely were 
included in the group, and others revealed during the 
course of the interview that they had applied for the 
award but not received it. Participants were asked to 
complete a one-page questionnaire which then formed the 
basis for an oral interview. 

Ill) Results. 

A) Analysis of Institutional Ratings. According to 
the IDEA I nterpr e t i ve Qu ide (Cashin, Brock, Owens, & 
Slawson, 1976) the standard error of measurement for most 
of the twenty categories Lb approximately 0.3. A 
comparison of the first semester's means (Fall 1984-85) 
with tlie seventh semester's means (Fall 1987-88) showed 
that positive changes in institutional b.eans fall within 
the range of the standard error of measurement for all 
twenty categories. It also should be noted that there 
was little negative movement in any of the categories. 
Institutional teaching effectiveness ratings generally 
remained constant during the seven semesters studied. 

B) Analysis of Individual Ratings. A review of the 
means and standard deviations based on the "similar 
course** comparison which takes into account class size 
and motivation level revealed some distinct patterns. 
Means and standard deviations for each of the IDEA 
summary outcome measures are contained in Tables 1.1 - 
1.7. 
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For all seven categories, the control group ratings 
(both pre--award and post--award) were much lower than the 
ratings of either the award recipients or non-- 
recipients. Differences in means between the award 
recipients and non--recipients were much less pronounced. 
In most cases, the award recipients had higher mean?, but 
that was not true for all categories. 

The tables also allow comparison of pre--award and 
post-- award means and standard deviations for each group. 
The largest difference is founi in Table 1.1, where the 
non- recipients rated 90.50 on the post--award as compared 
to 75.00 on the pre--award, a change of 15.50 percentile 
points. In most cases, the rating is higher in the post-- 
award category, but there were five exceptions to this. 
Both award recipients and the control group means 
declined slightly in '^Improved Attitude toward Field 
(Table 1.3), the non--recipients declined slightly in 
Communicating Content and Purpose (Teible 1.5) and both 
award recipients and the control group declined slightly 
in ''Preparing Examinations'* (Table 1.7). Most 
categories, however, indicate small increases from pre- 
award to post-- award. 

An analysis of covariance was conducted which 
compared the differences between groups before the award 
to differences within each group after the award. For 
this design, two analyses were possible: repeated 
measures analysis of variance 'or analysis of covariance. 
Since the groups were not randomly assigned, and since 
pre-existing differences were present in the groups, the 
preferred analysis was the analysis of covariance. This 
analysis answered the following question: "If the groups 
began at the same point (at the pre-test), would the 
groups differ at the post-test?" Results of the analyses 
of covariance are contained in Tables 2.1 - 2.7, 
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The "covariates" source in Tables 2.1 through 2.7 
indicates the amount of variability associated with the 
correlations between the pretest and the posttest. By 
itself^ this term does not answer a significant research 
question, and merely demonstrates that the assumptions 
for the an^lysiF of covariance have been met. For all 



seven categories, the covariates are significant 
(E > .001) . 

Variability between groups (award recipients, non- 
recipients, and control group) at the post-award is 
indicated by the source "main effects-group.** Of the 
seven categories in the tables, ** Improved Attitude toward 
Field** (Table 2.3) shows the most significant main effect 
(F ^ 3.70, df - 2, c - .032). However, in this case as 
in all seven cases, the results are much less significant 
than the differences among the three groups at the start. 
When significant differences were found, the post hoc 
test indicated that the awardees and the non-recipients 
were significantly higher than the controls. Tha results 
in all seven categories do not indicate that the award 
itself has had an effect on teacher effectiveness 
ratings . 

C ) Analysis of Qualitative Data • Initial 
qualitative data were drawn from a questionnaire 
completed by the inidividual faculty member prior to an 
oral interview. Table 3 contains the six questionnaire 
items, along with summary data of the original written 
responses • 
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Each individual was then asked to elaborate on 
his/her responses during the oral interview. Faculty 
were nearly unanimous in their belief that the quality 
of instruction at the college is improving. However, the 
ten faculty interviewed have doubts about whether the 
award impacts institutional teaching effectiveness, seem 
quite sure that the award has had no effect on their own 
teaching, and view the program as a reward system for 
past performance. Besides xhe fact that it is perceived 
as grounded in student evaluation, the program was 
criticized on the basis of faculty perceptions that: 'a) 
it is based on ambiguous rhetoric, (b) it is political, 
and (c) it has an inappropriate focus. There were 
diverse suggestions as to how instruction at the college 
might be improved using methods other than the Excellence 
in Teaching Program. 

IV) Discussion. 

The present study indicated that the introduction 
of a faculty incentive grant program has had no 
significant impact on teaching effectiveness at Messiah 
College. Statistical data for the institution showed 
that teaching effectiveness as measured on the IDEA form 



has remained nearly constant in the semesters since the 
program began* Analyses of teaching effectiveness 
measures for grant recipients, grant applicants who did 
not receive an award, and a control group indicated that 
the awards have been distributed to faculty who indeed 
are significantly more effective teachers as measured by 
the IDEA system. However, there was no indication of 
any change in teaching effectiveness as a result of the 
award for either successful applicants, unsuccessful 
applicants, or non-- applicants. Interviews with faculty 
corroborated the statistical conclusion, in that faculty 
have not perceived change in teaching effectiveness at 
either the institutional or individual level as a result 
of the incentive grant program. 

The basic assumption of faculty incentive grant 
programs seems to be that a reward will increase 
productivity. Programs then need to operate in full 
awareness of the research which has yielded some warnings 
about the use of rewards. For example, Deci and Ryan 
(1985) have referred to dozens of studies which indicate 
that the overuse of extrinsic rewards can decrease 
achievement levels. Adding an additional reward to an 
intrinsically enjoyable task may over justify the 
activity. If people sense that they are externally 
controlled, there is a potential for what was once 
enjoyeible to lose its appeal. 

On the other hand , there is evidence that the 
informative use of rewards and support may work to 
increase motivation (Pittman, Davey, Alafat, Vetherill, 
& Kramer, 1980). Part of the challenge of designing a 
reward program is to builrl a structure that is perceived 
as supportive and constructive rather than controlling. 
A further consideration in light of recent research is 
whether a reward program should be competitive. Spence 
and Helmreich (1983) conducted studies of intrinsic 
motivation which concluded that achievement is most 
effectively fueled by individual drives for subject 
mastery and hard work, and that achievement is least 
likely when instrinsic motivation is tied to 
competitiveness. Since faculty work is generally 
independent and sxibject-oriented, it may be counter- 
productive to allow competition to be perceived as an 
integral part of any developmental program. 

One set of numbers drawn from this study has some 
especially interesting implications . The largest 
difference between pre*>award and post-award was found in 
the "Overall Evaluation"* rating of unsuccessful 
applicants — those who did not receive an award in spite 
of the fact that they had IDEA evaluations which 
qualified them. While the statistical analyses serve as 
warning against putting much emphasis on this change, the 



increase represents the datum which is most supportive 
of the idea that grants might effect improved teaching. 
It it possible to explain this outcome as resulting from 
a competitive drive — to win the award in another year of 
application, or to prove oneself capable and deserving 
regardless of the award outcome. However, the 
possibility that the denial of an award has a greater 
motivating effect than the receiving of an award raises 
serious questions for those who design and administer 
award programs who undoubtedly would want to avoid 
appearances of coercion. 

It should be stressed at this point that this study 
dealt with the impact of the faculty incentive grant 
progreun on only one outcome, teaching effectiveness . 
While the results show quite clearly that the program has 
had little, if any, impact in that area, there is the 
possibility that it has had other effects. Since the 
award in this case is often tied to research projects, 
it would be reasonable to assess the award's impact on 
professional development outside the classroom. In that 
light, the Messiah College incentive grant program may 
reflect an institutional desire to increase publishable 
works by the faculty (while at the same time requiring 
that teaching remain above average). This is typical of 
a trend for "teaching- institutions to be moving toward 
greater research emphasis. Faculty who receive the award 
have a great amount of flexibility in the type of project 
they pursue, but the institution may gain credibility and 
prestige based on what that faculty member produces . 
Indeed, current faculty development literature proposes 
that "where development programs for teaching and 
research . . . exist separately, they should be merged. 
Then we can begin to think of a teacher-scholar 
profession, removed from the schizophrenia of 
teaching/research/service- (Young, 1987, p. 14). Lack 
of faculty enthusiasm about the incentive grant program 
at Messiah College suggests that attempts to merge 
institutional emphases on teaching and scholarship will 
need to proceed with care. 
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Table 1.1 

Means and Standard Deviations of Percentile Rank 
"Overall Evaluation" 



AWARD RECIPIENTS 
All Courses 
Similar Courses 

NON-RECIPIENTS 
All Courses 
Similar Courses 

CONTROL GROUP 
All Courses 
Similar Courses 



Pre-Award 
Mean Std. Dev. 

77.75 13.49 
78.75 12.32 

67.50 26.90 
75.00 17.21 

56.13 21.27 
58.98 18.94 



Post-Award 
Mean Std. Dev. 

83.08 13.99 
83.58 16.92 

85.07 22.40 
90.50 11.06 

61.57 23.39 
64.07 23.68 



Table 1.2 

Means and Standard Deviations of Percentile Ranks 
"Would Like Instructor Again" 



Pre-Award Post-Award 





Mean 


Std. 


Dev. 


Mean 


Std. 


Dev. 


AWARD RECIPIENTS 














All Courses 


75.54 


13. 


84 


77.21 


12. 


52 


Similar Courses 


75.25 


13. 


76 


79.92 


13. 


37 


NON-RECIPIENTS 














All Courses 


54.76 


26. 


91 


64.36 


28. 


90 


Similar Courses 


61.21 


24. 


93 


6^.38 


24. 


82 


CONTROL GROU'' 














All Courses 


46.43 


22. 


66 


51.58 


25. 


90 


Similar Courses 


44.40 


25. 


59 


48.17 


25. 


59 



Table 1.3 

Means and Standard Deviations of Percentile Ranks 
"Improved Attitude Toward Field" 

Pre-Award Post-Award 

Mean Std. Dev. Mean Sta . Dev . 

AWARD RECIPIENTS 

All Courses 72.33 17.24 67.71 23.19 

Similar Courses 74.29 15.73 72.75 16.51 

NON-RECIPIENTS 

All Courses 54.50 27.26 70.29 28.43 

Similar Courses 63.36 27.17 73.36 22.72 

CONTROL GROUP 

All Courses 50.83 22.51 53.07 25.42 

Similar Courses 48.75 20.61 47.95 22.93 



Table 1.4 

Means and Standard Deviations of P-.-'rcentile Ranks 
"Involving Students" 



AWARD RECIPIENTS 
All Courses 
Similar Courses 

NON-RECIPIENTS 
All Courses 
Similar Courses 

CONTROL GROUP 
All Courses 
Similar Courses 



Pre -Award 
Mean Std. Dev. 

78.33 15.65 
78.67 13.87 

66.64 35.30 
70.57 31.23 

53.00 23.23 
54.20 22.41 



Post -Award 

Mean Std. Dev. 

81.08 16.89 
79.29 18.51 

77.43 32.71 
76.00 31.18 

60.33 21.93 
60.53 18.16 



Tv'ibi.e 1.5 

Means and Standard Deviations of Percentile Ranks 
"Communicating Content and Purpose" 



Fre-Award Post-Award 





Mean 


Std. Dev. 


Mean 


Std. 


Dev. 


AWARD RECI7IENTS 












All Courses 


77.50 


11.43 


76.25 


13. 


31 


Similar Courses 


76.33 


10.54 


76.83 


14. 


41 


NON-RECIPIENTS 












All Co " rses 


57.71 


21.93 


63.57 


22. 


89 


Similar Courses 


67.93 


16.16 


66.29 


16. 


59 


CONTROL GROUP 












All Courses 


44.67 


18.90 


49.43 


21. 


27 


Similar Courses 


42.98 


' 17.76 


46.13 


20. 


01 



Table 1.6 

Means and Standard Deviations of Percentile Ranks 
"Creating Enthusiasm" 



AWARD RECI-iENTS 
All Courses 
Similar Courses 

NON-RECIPIENTS 
All Courses 
Similar Courses 

CONTROL GROUP 
All Courses 
Similar Courses 



Pre -Award 

78.25 18.70 

78.04 14.06 

68.79 31.26 

76.36 23.14 

45.77 19.43 

44.55 17.87 



Po8t->Award 

Mean Std. Dev. 

79.63 13.82 

78.42 14.07 

77.00 27.31 

77.00 25.45 

50.38 21.38 

47.38 22.18 



Table 1.7 

Means and Standard Deviations of Percentile Ranks 
"Prepeiring Examinations" 



AWARD RECIPIENTS 
All Courses 
Similar Courses 

NON-RECIPIENTS 
All Courses 
Similar Courses 

CONTROL GROUP 
All Courses 
Similar Courses 



Pre -Award 



56.17 
57.17 

61.43 
62.93 

49.35 
50.98 



19.03 
17.37 

18.41 
17.93 

26.05 
25.91 



Post-Award 
HfiAn Stdt Pevt 



57.17 
56.92 

66.79 
65.79 

43.47 
44.07 



14.76 
15.11 

16.95 
16.61 

25.58 
25.33 



Table 2.1 

Analysis of Variance of Post-Award Groups 
"Overall Evaluation" 



SOURCE SS df MS F P-value 
Covariates 5578.57 1 5578.57 13.90 .001 
Main Effects- 
Groups 2381.03 2 1190.52 2.96 .062 
Residual 18063.17 45 401.40 



Table 2.2 

Analysis of Variance of Post-Award Groups 
"Would Like Instructor Again" 

SOURCE SS df MS F P-value 
covariates 15419.39 1 15419.39 41.22 .001 
Main Effects- 
Groups 1235.25 2 617.63 1.65 .203 
Residual 16833.69 45 374.08 



Table 2.3 

Analysis of Variance of Post-Award Groups 
"Improved Attitude Toward Field" 

SOURCE SS df MS F P-value 
Covariates 6013.59 1 6013.59 13.93 .001 
Main Effects- 
Groups 3197.28 2 1598.64 3.70 .032* 
Residual 19420.23 45 431.56 



♦significant 



Table 2.4 

Analysis of Variance of Post-Award Groups 
"Involving Students" 



SOURCE SS df MS F P-value 
Covariates 10781.69 1 10781.69 41.32 .001 
Main Effects- 
Groups 268.74 2 134.37 .52 .601 
Residual 11741.77 45 260.93 



Table 2.5 

Analysis of Variance of Post-Award Groups 
"Conununicating Content and Purpose" 

SOURCE SS df MS F P-value 
Covariates 7706.42 1 7706.42 23.83 .001 
Main Effects- 
Groups 2149.36 2 1074.68 3.32 .062 
Residual 14555.42 45 323.45 



Table 2.6 

Analysis of Variance of Post-Award Groups 
"Creating Enthusiasm" 



SOURCE 
Covariates 
Main Effects- 
Groups 
Res idual 



SS 

18846.71 

216.59 
12106.20 



df 
1 

2 
45' 



MS 

18846.71 

108.30 
269.03 



F P-value 
70.06 .001 



.40 



.671 



Table 2.7 

Analysis of Variance of Post-Award Groups 
"Preparing Examinations" 

SOURCE SS df MS F P-value 
Covariates 6646.25 1 6646.25 16.97 .001 
Main Effects- 
Groups 1868.93 2 934.47 2.39 .103 
Residual 17620.67 45 391.57 



Table 3 

Faculty Questionnaire and Summary of written Responses 



1. Do you believe that the quality of instruction at this 
college has been improving? What are the reasons for 
this [lack of] change? 

[10 affirmative responses, with 3 of them somewhat 
qualified] 

2. Is your own teaching beccwiing nore effective? What 
makes you think so? 

[8 affirmative responses; 2 negative] 

3. What effect has the Excellence in Teaching Award 
program h id on teaching effectiveness at this college? 

[0 report positive effect; 6 report no effect; four see 
negative effect] 

4. Has the Excellence in Teaching Award program 
encouraged you to put more effort into classroom 
teaching? 

[2 report -possibly-; 8 negative responses] 

5. Do you perceive of the Excellence in Teaching award 
as a reward for past performance or as an incentive for 
improved future performance? 

[9 perceive -reward-; 1 perceives it as neither] 

6. What kinds of things would help you work toward 
becoming a more effective teacher? 

[varied respo^^ser \ 



